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ABSTRACT 



The Office of Research and Evaluation (ORE) of the 



Austin Independent School District (AISD) (Texas) reviews the 
effectiveness of many of the school district's special programs. In 
1992-93, ORE reviewed 60 programs or program components. Cost 
effectiveness was calculated for 31 programs, using an achievement 
effect measure for 10, a dropout prevention effect measure for 13, 
and a drug prevention effect measure for 8. An additional three 
dropout prevention programs were rated on effectiveness, although 
cost information could not be obtained. Three drug prevention 
programs were rated for cost, but not effectiveness, and another 23 
programs were evaluated on effectiveness based on other evaluation 
information. Two-thirds of the programs were rated as effective, and 
607. were considered cost-effective where calculation was possible. 
Three of four elementary school technology schools were the most 
cost-effective among achievement improvement programs. A 
parent-training program, Mega Skills, was the most cost-effective 
dropout prevention program, while Plays for Living and Drug Abuse 
Resistance Education (DARE) were the most cost-effective drug abuse 
prevention programs. Seventeen tables present evaluation findings. 
(Contains 11 references.) (SLD) 
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PROGRAM EFFECTIVENESS IN AISD, 1992-93 

Executive Summary 

Autiiors: David Wilkinson, Evangelina Mangino 



Austin Independent School District 
Office of Research and Evaluation 

Program Description 

For the past five years, ORE has provided the 
Board of Trustees with comparisons of the 
effectiveness of many of the District's special 
programs. Beginning in 1992-93, in response to 
the Board's request to provide it with a measure 
of effect as well as cost for the programs 
examined, ORE has prepared program 
effectiveness charts which include, where 
possible, cost-effectiveness for the programs and 
program components evaluated. Each February, 
at the Board's annual budget study session, ORE 
presents these program effectiveness charts for 
the programs evaluated the previous year. The 
document presented to the Board in February 
1993 was a working draft. This report is the 
finished product. 

Cost-effectiveness was calculated by dividing a 
measure of cost in dollars by one of three 
measures of effect: (1) achievement, (2) not 
dropping out, or (3) not using drugs. The cost 
of a program was defined as a program's 
appropriation (i.e., budget). The achievement 
measure of effect was based on standardized test 
scores from either the Norm-referenced 
Assessment Program for Texas (NAPT) or the 
Iowa Tests of Basic Skills (ITBS). The dropout 
prevention measure of effect was derived from 
the comparison in ORE's GENcric Evaluation 
SYStem (GENESYS) of the number of students 
in a program predicted to drop out with the 
actual number who did drop out. The drug 
prevention measure of effect was determined 
from the comparison of the rate of recent use of 
an illicit substance by program participants with 
the rate of recent use by students in the District 
overall. The cost-effectiveness ratio, expressed 
in dollars, which results from dividing cost by 
effect (C/E) is a measure of the cost- 
effectiveness of a program, i.e. the amount of 
effect for monies expended. 

Where cost or effect measures were not 
obtainable, and other evaluation information 
about a program was available, ORE staff 
assigned ratings of effectiveness to the programs 
evaluated based on other indicators, such as 
survey results or the attainment of program 
objectives. 



Major Findin gs 

1. ORE reviewed 60 1992-93 programs or 
program components. Cost-effectiveness was 
calculated for 31 programs (52%), 10 using 
an achievement effect measure, 13 using a 
dropout prevention effect measure, and 8 
programs using a drug prevention effect 
measure. An additional three dropout 
prevention programs were rated on 
effectiveness, although cost information could 
not b<; obtained. The coats of three drug 
prevention programs were obtained, but 
effectiveness information was lacking. 
Another 23 programs were rated on 
effectiveness based on other evaluation 
information. (Pages 6-18) 

2. From the review of program, it was 
determined that: 

• Two thirds (65%) of the programs were 
rated as effective; 60% were cost-effective 
(of those where calculations were 
possible). (Pages 6-18) 

• Three of the four elementary technology 
schools were the most cost-effective 
among achievement improvement 
programs. (Pages ii, 7-8) 

• MegaSkills, a parent training program, 
was the most cost-effective dropout 
prevention program. (Pages ii, 10-11) 

• Plays for Living and DARE were the most 
cost-effective drug abuse prevention 
programs. (Pages ii, 13) 



Budset Implications 

Mandate: Requested by the Board of Trustees 

Funding Amount: $10,000 (estimated) (for 
producing the program effectiveness report) 

Funding Source: Local 

Implications: 

While still developmental, the methodology for 
assessing the cost-effectiveness of programs 
provides additional perspective not afforded by 
separate assessments of effectiveness and cost. 
Programs can be evaluated in terms of their 
relative costs in meeting the same outcome 
criteria: improving student achievement, 
preventing students from dropping out, or 
preventing students from using drugs. In other 
words, alternative programs can be evaluated on 
the basis of their costs for raising student test 
scores, or the cost for each potential dropout 
averted, or the cost for each student prevented 
from alcohol or other drug use. Other success 
indicators notwithstanding, information about 
which programs provide the maximum 
effectiveness per level of cost cr require the 
least cost per level of effectiveness will assist in 
decisions about which programs to keep and 
expand, which to modify, and which to 
discontinue. 
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1992-93 Programs, Ranked Ordered According to Cost-Effectiveness, Most to Least 



PROGRAM (Based on an Achievement Measure) 


COST/EFFECT 


Elementary Technology Demonstration Schools - Pat ton 
Funding Source: External Grades: Pre-K - 5 


> CO 


Elementary Technology Demonstration Schools - Galindo 
Funding Source: External Grades: Pre-K - 5 


$ 67 


Elementary Technology Demonstration Schools ~ Andrews 
Funding Source: External Grades: Pre-K - 5 


% 1 19 


Science Acsdemy 

Funding Source: External Grades: 9-12 




Funding Source: External Grades: K-6 


t 1 77 
9 lit 


Chipter 1 Supplementary Instruction (low achievers) 
Funding Source: External Grades: K-6 


% 411 


Priority Schools (low achievers) 

Funding Sour :e: External & Local Grades: K-6 


$ 463 


PROGRAM (Based on a Dropout Prevention Measure) 


COST/EFFECT 


MegaSkills (High Schools) Funding Source: External Grades: 9-12 


$ 517 


Mltnlnr I\tiAA\n C/»hf\sxflA 

ivicmor ^iviiuuic ocnooisj 

Funding Source: Externa Grades: 7-8 




Newcomers Program ^Titlc VII) 

Funding Source: External Grades: 9-12 


$ 984 


rw- irviiuuie ocnooisj 

Funding Source: External Grades: 9-12 


> 1 ,043 


Robbins Secondary School 

Funding Source: External Grades: 9-12 


$ 3,978 


Mentor (High School) 

Funding Source: Externa; Grades: 9-12 


$ 4,853 


Austin Youth River Watch 

Funding Source: External Grades: 9-12 


$ 61,050 


ESOS (Martin and Mendez) 

Funding Source: External Grades: 7-8 


$ 68,229 


SHIRY (Kealing JHS) 

Funding Source: External Grade: 7 


$ 106,200 


ESOS (Johnston) Funding Source: External Grades 9-12 


$ 250,743 


PROGRAM (Based on a Drug Prevention Measure) 


COST/EFFECT 


Plays for Living 

Funding Source: External Grades: Prc-K - 6 


$ 13 


rtma Ahiitu* !2f»*t*f »nf^» F/inc^linn ^riADP^ 

Funding Source: External Grades: 5,7 




(ViegAoKlllS 

Funding Source: External Grades: Pre-K- 12 




Pti Af A ftBi tf an/ 1 ^ an/i 1 />lnj> rin iri / LM All 

rccr nsiisiance •no i^caucrHiip \ri\i*j 

Funding Source: External Grades: Prc-K- 12 




Innovative Programs 

Funding Source: External Grades: K-12 


$ 1.000 


Student Alcohol and Drug Abuse Education and Prevention Program (SADAEPP) 
Funding Source: External Grades: 4-12 


S 1 ,259 


Quality Schools 

Funding Source: External Grade: S 


$ 3,820 



Austin Independent School District 
Office of Research and Evaluation 
1992-93 ROSE Summary Rank Order 



HIGH SCHOOLS 


JR. HIGH/MIDDLE SCHOOLS 


ELEMENTARY SCHOOLS 


Austin 


0.25 


Kealing 


0.19 


Maplewood 


0.48 


L.B.J. 


0.21 






Bryker woods 


0.35 










Brooke 


0.32 










Mathews 


0.32 










Zavala 


U.2V 










Barrington 


0.27 










Gullett 


0.25 










Highland Park 


0.25 










Walnut Creek 


0.21 










Casis 


0.20 










Lee 


0.18 










Ortega 


0.17 


Anderson 


0.12 


Burnet 


0.08 


Barton Hills 


0.14 










Houston 


0.14 










Under 


0.1.2 










Doss 


0.09 










Oalindo 


0.08 










Zilker 


0.08 










Allison 


0.08 










Pease 


0.07 










Travis Heights 


0.06 










Norman 


0.06 










Palm 


0.05 










Andrews 


0.05 










Blanton 


0.05 


Johnston 


0.04 


Fulmore 


0.05 


Oak Hill 


0.04 


Lsnier 


-0.01 


Martin 


0.04 


Campbell 


0.04 


McCallum 


-0.03 


O.Henry 


0.04 


Patton 


0.04 






Lamsr 


0.03 


Odom 


0.03 






Dobie 


0.02 


Woold ridge 


0.03 






Murchison 


0.00 


Joslin 


0.03 






Pcarce 


-0.01 


Pillow 


0.03 






Covington 


-0.01 


Blackshcar 


0.02 










Hill 


U.Ux 










Davis 


0.02 










Graham 


0.01 










Menchaca 


0.01 










Oak Springs 


-0.03 










Cook 


-0.04 










Sanchez 


-0.04 










Pleasant Hill 


-0.04 


Bowie 


-0.09 


Bcdichck 


-0.07 


Reilly 


-0.05 


Crockett 


-0.12 


Porter 


-0.09 


Cunningham 


-0,05 


Robbing 


-0.i4 


Webb 


-0.09 


Dawson 


-0.06 






Mcndez 




Harris 


-0.06 










Kikcr 


-0.06 










Brentwood 


-0.06 










Becker 


-0.07 










vv iucn 


-u.u / 










Jordan 


-0.08 










Pecan Springs 


-0.08 










Allan 


-0.10 










Brown 


-0.11 










St. Elmo 


-0.12 










Sunset Valley 


-0.13 










Woolen 


-0 13 


Travis 


-0.25 






Kocurek 


-0.15 


Reagan 


-0.27 






Winn 


-0.15 










Summitt 


-0.17 










Met/. 


-0.18 










Lang ford 


-0.19 










Govallc 


-0.22 










Ridgctop 


-0.23 










Williams 


-0.31 










Boone 


-0.32 










Sims 


-0.41 
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PROGRAM EFFECT IVENESS IN AISD, 1992-93 



Background 

For the past five years, ORE has provided the Board of Trustees with comparisons of the effectiveness of many 
of the District's special programs. In 1992-93, at the Board's request to provide it with a measure of effect as well 
as cost in the program effectiveness charts ORE prepares for the Board's annual budget study session, ORE 
conducted a retrospective examination of 1991-92 AISD programs. In February 1993, ORE presented the Board 
with program effectiveness charts which included cost-effectiveness ratios for many programs evaluated during 
1991-92. Comments and suggestions from the Board of Trustees, District staff, and community members were 
used to fine tune the methodology and the format in which the information was reported. A final report titled What 
Works, And Can We Afford It? Program Effectiveness in AISD, 1991-92 was issued in May 1993. 

The methodology developed for assessing the cost-effectiveness of programs was applied to the programs evaluated 
by ORE in 1992-93, and each of ORE's final reports contained a program effectiveness summary including, where 
possible, cost-effectiveness information. In February 1994, ORE presented the Board with a draft document 
containing program effectiveness charts, and cost-effectiveness ratios, for programs evaluated during 1992-93. This 
report is the finished product. 

Cautions 

The methodology ORE has developed is still being refined. Although we have had numerous indications that we 
are on the "leading edge" of this type of analysis, we are mindful of several methodological difficulties which we 
have not fully resolved. Therefore, we do not represent our findings as the last word on how to determine what 
an effective program is. We hope, however, that they will provide a basis for continuing discussion about how 
best to evaluate the success of the District's programs. 

Please keep three factors in mind when interpreting cost-effectiveness ratios. 

1. Only achievement test scores, dropout rates, and self-reported drug and alcohol usage rates were used as 
measures of program effectiveness for calculating the cost-effectiveness ratios. Over the years, ORE has 
encouraged everyone to consider a wide range of information when assessing the impact of programs. For 
the purpose of calculating cost-effectiveness, however, what was needed were measures of effect common 
across all types of programs. Standardized achievement test scores, from the Norm-referenced Assessment 
Program for Texas (NAPT) and the Iowa Tests of Basic Skills (ITBS), were used because they our most 
reliable, broadest based, and most readily available measure of achievement. Other effectiveness measures 
need to be explored, however. For example, the elementary technology demonstration schools have shown 
better gains on the Texas Assessment of Basic Skills (TAAS) than on the NAPT. Readers are encouraged to 
read the detailed ORE evaluation reports to find information on other outcomes such as this. 

2. The methodology used to calculate the cost-effectiveness ratios, while based in the general research literature, 
was applied according to our best professional judgment. Much additional discussion and study of the 
methodology will need to occur to establish our confidence that it appropriately reflects how much effect is 
attained for each dollar spent in special programs. 

3. Better documentation and reporting of the costs of special programs and the numbers of students served is 
needed ii ^ost-effectiveness analysis is to be a wholly useful tool for evaluation and decision making. 
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Method 

Foilowing Henry Levin's definition of cost effectiveness, cost effectiveness is obtained by dividing cost by effect. 

cost 

CoX EfTectiveneu = cfTect 



The equation is very simple, but assigning values to the terms in the numerator and denominator is complex and 
can be controversial. 



Cost 

Program costs are reported as budgeted amounts. Actual expenditures may vary. Some programs with relatively 
low costs may require substantial indirect resources for staff support, facilities, etc. Volunteers hold the costs down 
in some programs, but expansion of those programs could cost more if the pool of available volunteers is not large 
enough to accommodate expansion. 

Outcomes or Effect 



Program outcomes in these 
charts are again simple in 
concept, though more 
complicated in application. If 
available, NAPT/ITBS scores 
were used. If the program 
fc\ised on dropout prevention, 
then the dropout rate was used. 
If the program focused on drug 
abuse prevention, then student 
drug use rate was employed. 



ACHIEVEMENT 


EFFECT 
NOT DROPPING OUT 


NOT USING DRUGS 


ITBS/NAPT GAIN 

vs 

Comparable Students 
or 

District Average 
or 

National Average 


Difference between the number of 
students who were predicted to 
drop out and the actual number of 
dropouts 


Difference between the 
number of program 
students using drugs 
and the number using 
drugs districtwide 



This procedure seems straight- 
forward, but NAPT/ITBS is 
only one of many measures of 
student academic progress. 

TAAS, college entrance exams (SAT and ACT), grade-point average (GPA), and many other alternatives could 
be used. NAPT/ITBS was chosen because it is our most reliable, broadest based, and most readily available 
measure. In order to compare cost-effectiveness across programs, a single effect measure is essential. 

Where NAPT/ITBS are used, outcomes are reported as the achievement gain in grade equivalent months-above 
and beyond what the students would have gained without the program. A grade equivalent month is the amount 
of gain made on the NAPT/ITBS by an average student during one month of instruction. 

For programs for at-risk students, clearly the dropout rate is appropriate. However, these programs can certainly 
have benefits beyond just keeping students in school. These charts look simply at how much the program spent 
to keep oji§ student from dropping out. In other words, if the student population served typically has 20 dropouts 
annually, and among the program students only 15 dropped out, then the program is credited with keeping five in 
school. This can make the cost per student kept in school high, because 20 at-risk students may have to be served 
to net one dropout kept in school. 



92.41 



For drug prevention programs, rates of student drug use are appropriate outcomes. However, these rates must rely 
on the anonymous responses of a sample of students surveyed about their own use of illicit substances. In addition, 
student identification of the programs serving them many not have been flawless. 

Programs for which no NAPT/ITBS were available and which were not dropout or drug prevention programs were 
rated on the basis of other evaluation information collected. 



Cost-effectiveness 

Outcomes are divided into the cost of 
the program per student to give the 
cost to produce one month of 
achievement gain, or into the total 
program cost to calculate the cost to 
keep one potential dropout in school, 
or into the total program cost to 
calculate the cost to prevent one student 
from using drugs. A caution to the 
reader is that we may not be able to 
produce twice the effect for twice the 
cost. We do not know what 
relationships would exist if we spetit 
more or less money on a program. 
However, this cost-effectiveness 
number does tell us what we did spend 
for the amount of effect realized. 

Some programs do not have a cost- 
effectiveness amount shown, because 
they had no positive effect or because 
their impact was actually negative. 



COST/EFFECT (C/E) 



Cost/ Achievement Gain 
or 

Cost for at-risk students/*? of potential dropouts staying in school 

or 

Cost/# of students r.ot using drugs 



RATING 


Surveys: 


Other Indicators: 


-Staff 


- Retention Rate 


- Students 


- Attendance 


- Parents 


- Goal Attainment 




- Discipline Rate 
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Effectiveness of 1992-93 Programs 



Attached are several charts showing the cost-effectiveness of some of AISD's 1992-93 special programs. 

A. Cost-effectiveness of 1992-93 Programs Based on an Achievement Measure 

B. Cost-effectiveness of 1992-93 Programs Based on a Dropout Prevention Measure 

C. Cost-effectiveness of 1992-93 Programs Based on a Drug Prevention Measure 
Another chart shows program effectiveness where cost-effectiveness could not be calculated. 

D. Effectiveness of 1992-93 Programs Based on Other Indicators 

Effectiveness of Schools 

In many ways, schools may be thought of as programs and are sometimes the more appropriate unit of analysis. 
The Report On School Effectiveness (ROSE) serves as the basis for comparing the effectiveness of schools. If the 
differences between predicted and actual achievement in each test area in each grade in a school, expressed in grade 
equivalents, are averaged, the result is the average residual (difference) for the school. This statistic is presented 
in the attached chart for all AISD schools for 1992-93. The schools are ordered from most positive difference to 
most negative difference. 
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Findings 

1992-93 Programs 

Most programs evaluated in 1992-93 in AISD are rated as effective. Approximately 21 % of the ratings are based 
on achievement, 35% are based on the number of students not dropping out, and 44% are based on other evaluation 
findings. 



Achievement Gains 


Successful Academic Programs 


Effect 


Cost/Effect 




(Gain) 


Index 


In general, the programs showing the 


Science Academy 


12 


130 


highest achievement gains for students 


Ch.l Schoolwide Project (Low Achievers) 


2 


177 


served tend to be programs that offer 






students enriching experiences in 


Ch.l Supplementary (Low Achievers) 


2 


411 


addition to the regular curriculum. 


Technology at Patton 


1 


65 


Most of these programs have a 


Technology at Galindo 


! 


67 


relatively high initial cost. But once 


the program is in place, the gain for 


Technology at Andrews 


1 


119 


the per-pupil cost is relatively low. 


Priority Schools (Low Achievers) 


1 


463 











Successful Dropout Prevention 


Effect 


Cost/Effect 


A common feature among successful 


Programs 


(% Stayed) 


Index 


uropoui prevenuon programs is uiat 


MegaSkills (High School) 


100 


515 


thpv nrnviHp QtnHpnfc xi/itH inHiviHunl 


ESOS 


100 


68,400 


□ rr^nfrifln rsv tha r\ncc iV*il it\r flfavSKSlif't/ 

ducMuuii ui uic pubbiDiuiy ur ncxioimy 


SHIRY (Kealing) 


100 


106,667 


in class schedules and enrichment 


activities. Many of these programs are 


Project MAN 


100 


Not Available 


dependent on the use of volunteers or 


Titie VII Newcomers Program 


96 


9S4 


mentors. The cost reported for these 


96 




programs does not reflect the in-kind 


Robbins 


4,161 


contribution of volunteers. 


Peer Assistance and Leadership (PAL) 


50 


1,637 




Project Mentor (Middle/Jr. High School 


40 


684 




Austin Youth River Watch Program 


33 


61,210 




ESOS (Johnston) 


33 


250,800 










Drug Prevention 


Successful Drug Prevention Programs 


Effect 


Cost/Effect 




(% Prcvcn!«D 


Index 


Among successful drug prevention 


MegaSkills 


17 


200 


programs, underlying themes are the 


Drug Abuse Resistance Education (DARE) 


10 


39 


presentation of informative material 


Program 






and the interaction with other caring 


Plays for Living 


10 




persons-parents, police officers, and 


13 


older peers. Drug prevention programs 


Innovative Programs 


4 


1,000 


which serve all of the students at a 


Peer Assistance Leadership (PAL) 


4 


942 


specific grade (e.g., DARE) cost less 






than programs that select students 


Quality Schools 


3 


3,880 


based on specific characteristics. 


Student Alcohol and Drug Abuse Education 


3 


1,259 




and Prevention Program (SADAEPP) 
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COST-EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON AN ACHIEVEMENT MEASURE 



EXAMPLE 



PROGRAM 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 
STUDENT 


EFFECT 
(in months) 


COST PES 
STUDENT rllR 1 
MONTH GAiN 
(COST/EFFECT) 


RATING 


Elementary Technology 
Demonstration School (Andrews) 

Funding Source: External 

Grades: Pre-K - 5 


$63,253 

$1,580,956 

Investment cost for 
hardware, software, 
and wiring. 


843 


S75 


R: 0.5 
M: 0.75 
Avg.: 0.63 


$119 


0 



Elementary Technology Demonstration School (Andrews), 1992-93 - Grades: Pre-K - 5 
Cost: $63,253 (1992-93 operations), $1,580,956 (hardware, software, and wiring). 
Number of Students Served: 843 

Cost Per Student: $75 [$63,253/843 = $75.03 = $75 rounded] 
Effect: R: 0.5 M: 0.75 Avg. = 0.63 

[Because all grades were served, the ROSE residuals in reading for grades 2, 3, 4, and 5 were averaged: -0.1 + 
0.2 + -0.1 + 0.2 /4 = 0.05. The mathematics ROSE residuals for grades 2, 3, 4, and 5 were averaged: -0.1 + 
0.1 + 0.2 + 0.1 /4 = 0.075. Effects are transformed to months by multiplying by 10, so the reading and 
mathematics effects become 0.5 and 0.75, respectively. The reading and mathematics ROSE residuals were 
averaged: 0.5 + 0.75 = .0.63.] 

Cost/Effect: $119 [$75/0.63 = $1 19.05 = $1 19 rounded] 

What this means is that it costs $1 19 per year per Andrews student using the computers to attain six-tenth of one 
month's achievement gain above that the student would normally have, achieved as the result of the regular 
instructional program. 

Rating: 0 (A rating of zero was assigned because less than a one-month achievement gain was made.) 
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Cost-Effectiveness of 1992-93 Programs Based on an Achievement Measure 



PROGRAM 


COST 


NUMBER OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 

Or>nt inn 

SERVED 


EFFECT 
fin months) 


COST 
EFFECT 


RATING 


Chapter 1 Schoolwide Projects (all 
students) 

Funding Source: External 
Grades: K-6 


$1,881,525 


4,633 


$406 


Reading: -1.0 
Math: N/A 
Avg.: N/A 






Chapter 1 Schoolwide Projects 
(low achievers) 

Funding Source: External 

Grades: K-6 


$616,308 


1,518 


$406 


Reading: 2.3 
Math: N/A 
Avg.: N/A 


$177 


+ 


Chapter 1 Supplementary Instruction 
(low achievers) 

Funding Source: External 

Grades: K-6 


$1,452,917 


1,682 


$864 


Reading: 2.1 
Math: N/A 
Avg.: N/A 


$411 


+ 


Elementary Technology 
Demonstration Schools - Andrews 

Funding Source: External 

Grades: Pre-K - 5 


$63,253 

$1,580,956 
Investment cost 
for hardware, 
software, and 
wiring. 


843 


$75 


Reading: .5 
Math: .75 
Avg: .63 


$119 


0 


Elementary Technology 
Demonstration Schools - Galindo 

Funding Source: External 

Grades: Pre-K - 5 


$44,235 

$246,000 
Investment cost 
for hardware, 
software, and 
wiring. 


751 


$59 


Reading: .5 
Math: 1.25 
Avg: .88 


$67 


0 


Elementary Technology 
Demonstration Schools - Langford 

Funding Source: External 

Grades: Pre-K - 5 


•S53.744 

$1,229,642 
Investment cost 
for hardware, 
software, and 
wiring. 


574 


S94 


Reading: -1.0 
Math: -1.5 
Avg: -1.25 




0 



Rating is expressed as contributing to any of the five 
AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown 



ERIC 



Cost is the expense over the regular District per 
student expenditure of about $4,000. 

0 No cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 
SSS Major direct costs for teachers, JUff, and/or 
equipment in the range of $500 per student 
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PROGRAM 


COST 


NUMBER OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


EFFECT 
(in months) 


COST 
EFFECT 


RATING 


Elementary Technology 
Demonstration Schools -Patton 

Funding Source: External 

Grades: Pre-K - 5 


S63.253 

$1,834,320 
Investment cost 
for hardware, 
software, and 
wiring. 


1,307 


$48 


Reading: .25 
Math: 1 .25 
Avg: .75 


$65 


0 


Priority Schools (all students) 
Funding Source: External & Local 
Grades: K-6 


$2,149,744* 


6,628 


S324 


Reading: -1.1 
Math: N/A 
Avg.: N/A 






Priority Schools (low achievers) 
Funding Source: External & Local 
Grades: K-6 


$380,052* 


1,173 


S324 


Reading: 0.7 
Math: N/A 
Avg.: N/A 


$463 


+ 


Science Academy 
Funding Source: Local 
Grades: 9-12 


$821,999 


546 


51,505 


Avg.: 11.5 


$130 


+ 
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COST-EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON A DROPOUT PREVENTION 
MEASURE 

EXAMPLE 



PROGRAM 


ALLOCATION 
(COST) 


NUMBER OF 
STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


NUMBER OF 
DROPOUTS 

Predicted Obtakwd 


PREDICTED 
DROPOUTS 
WHO STAYED 
IN SCHOOL 
(EFFECT) 

# % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


RATING 


Newcomers Program 
Funding Source: External 
Grades: 9-12 


$26,000 


At risk: 1 34 
Total: 134 


$940 


134 6 


123 96 


$984 


+ 



Newcomers Program, 1992-93 - Grades: 9-12 



Cost: $26,000 

Number of Students Served: 134 

Cost Per Student: $940 [$26,000/134 = $940.30 = $940 rounded] 
Effect: 128 

[Predicted 134 students, Obtained 6 students] 

134 - 6 = 128 students prevented from dropping out 

Cost/Effect: $984 [$26,000/128 = $984.38 = $984 rounded] 

What this means is that it costs $984 for each student prevented from dropping out by the Newcomers Program 
who would otherwise have been expected to drop out of school. 

Rating: + 
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Cost-Effectiveness of 1992-93 Programs Based on a Dropout Prevention Measure 



PROGRAM 


COST 


NUMBER 

OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


NUMBER OF 
DROPOUTS 
Predicted Obtained 


PREDICTED 
DROPOUTS 

WHO 
STAYED IN 
SCHOOL 
(EFFECT) 
» % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


RATING 


Austin Youth River Watch 
Funding Source: External 
Grades: 9-12 


At risk: $61,050 
Total: $82,500 


At risk: 23 
Total: 31 


$2,661 


3 2 


1 33 


$61,050 




Education for Parenthood 
Infant Development 
Centers (Johnston HS) 

Funding Source: External 

Grades: 


At risk: $44,100 
Total: $67,846 


At risk: 24 
Total: 37 


$1,834 


5 5 


0 0 




0 


ESOS (Johnston HS) 
Funding Source: External 
Grades: 9-12 


At risk: $250,743 
Total: $302,100 


At risk: 44 
Total: 53 


$5,700 


3 2 


1 33 


$250,743 


+ 


ESOS (Middle School*/ 
Martin and Mendez) 

Funding Source: External 

Grades: 7-8 


At risk: $ 68,229 
Total: $119,700 


At risk: 12 
Total :21 


$5,700 


1 0 


1 100 


$ 68,229 


+ 


MegaSkills (Middle 
Schools) 

Funding Source: External 
Grades: 7-9 


At risk: $1,786 
Total: $3,369 


At risk: 19 
Total: 36 


$94 


2 4 


0 0 




0 


MegaSkills (High Schools) 
Funding Source: External 
Grades: 9-12 


At risk: $1,033 
Total: $2,152 


At risk- 1 1 
Total: 23 


$94 


2 0 


2 100 


$ 517 


+ 


Mentor (Middle School) 
Funding Source: External 
Grades: 7-8 


At riak: $2,755 
Total: $4,750 


At risk: 144 
Total: 250 


$19 


10 6 


4 40 


S 689 


+ 



Rating is expressed as contributing to any of the five 
AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown 
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Cost is the expense over the regular District per 
student expenditure of about $4,000. 



0 
$ 

$$ 
$$$ 



7F 



No cost or minimal cost 

Indirect costs and overhead, but no separate budget 
Some direct costs, but under $500 per student 
Major direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 
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PROGRAM 


COST 


NUMBER 

OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


NUMBER OF 
DROPOUTS 
Predicted Obtained 


PREDICTED 

WHO 
STAYED IN 
SCHOOL 
(EFFECT) 
# % 


POST PFH 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


RATING 


Mcnlor (High School) 
Funding Source: Externa! 
Grades: 9-12 


At risk:$ 14,559 
Total: $33,858 


At risk: 774 
Total: 1,782 


$19 


202 199 


3 1 


$ 4,853 


+ 


Newcomers Program 
(Title VII) (High Schooli) 

Funding Source: External 

Grades: 9-12 


At riak:$ 126,000 
Total: $126,000 


At risk: 134 
Total: 134 


$940 


134 6 


128 96 


$984 


+ 


PAL (Middle Schools) 
Funding Source: External 
Grades: 9-12 


At risk: $3,286 
Total: $9,664 


At risk: 62 
(estimate) 

Total: 183 


$53 


4 2 


2 50 


$ 1,643 




Project MAN (LEI HS) 
Funding Source: External 
Grades: 9-12 


No financial 
information 
available 


At risk: 12 
Total: 25 




2 0 


2 100 




+ 


Robbins Secondary School 
Funding Source: External 
Grades: 9-12 


At risk. 36 
Toul: $1,081,936 


At risk: 272 
Total: 272 


$3,978 


272 12 


260 96 


$4,161 


+ 


SHIRY (Kealing JHS) 
Funding Source: External 
Grades: 7 


At risk: $106,200 
Toul: $180,000 


At risk: 16 
Total: 27 


$6,667 


1 0 


1 100 


$106,200 


+ 


SHIRY (Martin JHS) 
Funding Source: External 
Grades: 7 


At risk:$ 84,600 
Total: $180,000 


At risk: 8 
Total: 17 


$10,588 


1 1 


0 0 




0 


Zenith (Austin HS) 
Funding Source: External 
Grades: 9-12 


No financial 
information 
available 


At risk: 27 
Total: 32 




7 3 


4 57 






Zenith (Bowie HS) 
Funding Source: External 
Grades: 9-12 


No Mnancial 
information 
available 


At risk: 20 
Total: 34 




3 4 


-1 -33 







9 

ERIC 



1 



92.41 

COST-EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON A DRUG PREVENTION MEASURE 



EXAMPLE 



DRUG-FREE 
SCHOOLS (DFS) 
PROGRAM 


COST 


NUMBER OF 

STUDENTS* 

SERVED 


COST PER 
STUDENT* 


NUMBER OF STUDENTS 
PREVENTED FROM 
ALCOHOL AND OTHF.R 
DRUG (AOD) USE 
(EFFECT) 


COST PER 
STUDENT 

PREVENTED FROM 
AOD USE 
(COST/EFFECT) 


RATING 


Drug Abuse Resistance 
Education (DARE) 


$43,298 


11,190 


$3.87 


1,119 


$39 


+ 



* Participants 



Cost: $43,298 

Number of Students Served: 11,190 

Cost Per Student: $3.87 ($43,298/11,190 = $3.87) 

Number of Students Prevented from Alcohol and Other Drug (AOD) Use (Effect): 1,119 Students 
prevented from AOD use by the District Drug Free School (DFS) programs is calculated by 
subtracting the rate of use for the DFS program students (the recent use rate for DARE participants 
was 3C%), from the average rate of use for all students in the District (40%). That difference (10%) 
was multiplied by the total number of students served by the program (11,190 * .10 = 1,1 19) 

Number of Students Prevented from 

Alcohol and other Drug (AOD) Use (Effect): 1 , 1 19 

Cost Per Student Prevented 

from AOD Use (Cost/Effect): $39 ($43,298/1,119 « $38.69 = $39 rounded) 
Rating: + 



Rating is expressed as contributing to any of the five 
AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown 
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Cost is the expense over the regular District per 
student expenditure of about $4,000. 

0 No cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct cost 1 ;, iut under $500 per student 
$$S Major direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 

4.-8 
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Cost-Effectiveness of 1992-93 Programs Based on a Drug Prevention Measure 



DRUG-FREE 
SCHOOLS (DFS) 
PROGRAM 


COST 


NUMBER OF 

STUDENTS* 

SERVED 


COST PER 
STUDENT* 


NUMBER OF STUDENTS 
PREVENTED FROM ALCOHOL 
AND OTHER DRUG (AOD) USE 
(EFFECT) 


COST PER STUDENT 
PREVENTED FROM AOD 
USE (COST/EFFECT) 


RATING 


All Well Health Services 
Funding Source: External 
Grades: 


$ 3,000 


10 staff 


$300 staff 


Insufficient information 






Conflict Resolution 
Project 

Funding Source: External 
■ Grades: 


$33,352 


39 students 
57 staff 


$368 per 
participant 


0 






Drug Abuse Resistance 

Education (DARE) 

Funding Source! Extern*] 
Grades: 


$43,298 


11,190 


$3.87 


1,119 


$39 


+ 


Innovative Programs 

¥•1 1 rtfii n or Q nitre* f*Yf**mfli 

1 UJIUlll£ OUUIVVi LiALCUWl 

Grades: 


$37,014 


932 


$39.71 


37 


$1 000 




■v-12 Curriculum 
Funding Source: External 
Grades: 


$47,186 


64,171 


$ .74 


Rating based on program records of 
service 




+ 


Medicine Education and 
Safety Program 

Funding Source: External 

Grades: 


$ 5,772 






Rating based on completion of 
project and on reactions to 
conference presentations 




+ 


MegaSlcills 

Funding Source: External 
Grades: 


$21,798 
DFS 

$17,664 
Ch.2; 

$20,705 
Ch.i 


1,643 parents 
643 students 


$13.27 per 
parent DFS 
$93 .57 per 
student DFS 

$36.62 
(Total for all 
programs) 


109 


$200 


+ 


Peer Assistance and 
Leadership (PAL) 

Funding Source: External 

Grades: 


$56,715 


1,044 


$52.81 


42 


$942 


+ 


Plays for Living 
Funding Source: External 
Grades: 


$ 6,000 


4,472 


$1.34 


447 


$13 


+ 
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DRUG-FREE 
SCHOOLS (DFS) 
PROGRAM 


COST 


NUMBER OF 

STUDENTS* 

SERVED 


COST PER 
STUDENT* 


NUMBER OF STUDENTS 
PREVENTED FROM ALCOHOL 
AND OTHER DRUG (AOD) USE 
(EFFECT) 


COST PER STUDENT 
T'REVENTED FROM AOD 
USE (COST/EFFECT) 


RATING 


Private Schools 
Funding Source: External 
Grades: 


$18,143 


2,779 


$6.53 


Evaluation did not take place for 
this component. 






Quality Schools 
Funding Source: External 
Grades: 


$160,452 


306 staff 
602 students 


$524.35 per 
staff trained 
$266.53 per 
student 


42 


$3,820 


+ 


Student Alcohol and Drug 
Abuse Education and 
Prevention Program 
(SADAEPP) 

Funding Source: External 
Grades: 


$94,433 
DFS 

$20,579 
Ch.2 


2,488 


$37.96 


75 


$1,259 


+ 


Student Assistance 
Program (SAP) 

Funding Source: External 

Grades: 


$24,851 


185 staff 


$134.33 


Staff training was not evaluated this 
year. 







* Participants 



Rating is expressed as contributing to any of the five 
A1SD strategic objectives. 
< 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown 
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Cost is the expense over the regular District per 
student expenditure of about $4,000. 

0 No cost or minimal cost 

$ indirect costs and overhead, but no separate budget 
$$ Some direct cask, but under $500 per student 
S$S Major direct costs for teachers, staff, atid/or 
equipment in the range of $500 per student 
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EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON OTHER INDICATORS 

EXAMPLE 



PROGRAM 


COST 


NUMBER OF 

STUDENTS* 

SERVED 


COST PER 
STUDENT* 


EVIDENCE 


COST/EFFECT 


RATING 


Chapter 1 Migrant 
Supplementary Instruction 

Funding Source: External 

Grades: K-12 

Level of Service: 1-2 
hrs./week, all year 


$101,015 


124 


$815 


Program met its goals 




+ 



* Participants 
Cost: $101,015 

Number of Students Served: 124 

Cost Per Student: $815.00 ($1™ 015/124 = $815) 

Rating: + 
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Effectiveness of 1992-93 Programs Based on Other Indicators 



PROGRAM 


COST 


NUMBER OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


EVIDENCE 


COST/ 
EFFECT 


RATING 


Chapter 1 Migrant Supplementary 
Instruction 

Funding Source: External 
Grades: K-12 


5101,015 


124 


5815 


Program met 
its goals 




+ 


Chapter 1 Neglected of Delinquent 
Institutions 

Funding Source: External 
Grades: 1-12 


$109,768 


1,185 


$93 


Program met 
its goals 




+ 


Chapter 1 Nonpublic Schools 
Funding Source: External 
Grades: 1-7 


$26,608 


48 


$554 


Program met 
its goals 




+ 


Chapter 2 Academic Decathlon 
Funding Source: External 
Grades: 11-12 


$41,747 


70 


$487 


Rating based 
on staff surveys 




+ 


Chapter 2 Library Resources 
Funding Source: External 
Grades: K-12 


$43,950 


69,440 


$0.61 


Rating based 
on staff surveys 




+ 


Chapter 2 Megaskills 
Funding Source: External 
Grades: 6-8 


$17,664 Ch.2 

$20,705 
Chapter 1 
$21,798 Drug 
Free Schools 


1 ,643 (parents) 


$10.75 (per 
parent) 

536.62 (Total 
for all 
programs) 


Rating based 
on dropout 
rate, retention, 
grades, 
attendance, & 
discipline 




0 


Chapter 2 Middle School Fellows 
Program 

Funding Source: External 
Grades: 6-8 


$25,708 


44 (staff) 


5135 


Rating based 
on written 
comments 
offered by 
participants 




+ 


Chapter 2 Multicultural/Special 
Funding Source: External 
Grades: Pre-K-12 


$11,000 


10,208 


SI. 00 


Rating based 
on user survey 




+ 



Rating is expressed as contributing to any of the five 
AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unijiown 
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Cost is the expense over the regular District per 
student expenditure of about $4,000. 

0 No cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 
$$$ Major direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 
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PROGRAM 


COST 


NUMBER OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


EVIDENCE 


COST/ 
EFFECT 


RATING 


Chapter 2 Prekindcigartcn 
Supplements 

Funding Source: External 


$123,834 


114 


$209 


Rating based 
on PPVT-R & 
TV1P gains 
from pre- to 
posttcst 




+ 


Chapter 2 Private Schools 
Funding Source: External 
Grades: Prc-K-1 


519,803 


3,039 


$6.17 


Based on rating 
of purchases 




+ 


Chapter 2 Reading Recovery Teacher 
Leader Training 

Funding Source: External 

Grades: 


S57.062 


1 (staff) 


$57,062* 


Rating based 
on interview 
with 

participants 




+ 


Chapter 2 Secondary Library 
Technology Support 

Funding Source: External 


SI 8,280 


21,937 


$0.80 


Rating 'ased 
on staff survey 




+ 


Chapter 2 Spanish Academy 
Funding Source: External 
Grades: 


$32,899 


295 (staff) 


$112 


Based on 
course 

evaluation by 
participants 




+ 


Chapter 2 Student Alcohol and Drug 
Abuse Education and Prevention 
Program 

Funding Source: External 
Grades: 5-12 


$20,579 Ch.2 

$94,433 Drug- 
Free Schools 


2,488 


$8.27 

$46.23 (Total 
for all 
programs) 


Rating based 
cn staff and 
student survey 




+ 


Chapter 2 Technology Learning 
Center at iohnston High 

Funding Source: External 

Grades: 9-12 


$16,534 


1,723 


$9.60 


No assessment 
conducted 






Chapter 2 Using Technology for 
Access to Problem Solving 

Funding Source: External 

Grades: 8 


No funds 
received 


4,921 


0 


Chapter 2 
Discretionary 
project; No 
assessment 
conducted 






Chapter 2 Wicat Computer Lab at 
Blanton Elementary 

Funding Source: External 

Grades: (Pre-K-6) 


$17,133 


481 


$35.62 


Rating based 
on teacher 
survey 




+ 
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PROGRAM 


COST 


NUMBER OF 
STUDENTS 
SERVED 


COST PER 
STUDENT 
SERVED 


EVIDENCE 


COST/ 
EFFECT 


RATING 


Full-Day Prekindergarten 
Funding Source: . External 
Grades: Prc-K 


Sl.596,615 


1,702 


$938 


Rating based 
on average 
gains from Fall 
to Spring 
compared to 
national 
average. 




+ 


SBI- All Campuses 
Funding Source: Local 
Grades: K-12 


$108,398 


69,440 


$2 






0 


Title II Workshops 
Funding Source: External 


$27,242 


321 


C85 


Rating based 
on participant 
survey 




+ 


Title U Conferences 
Funding Source: External 
Grades: K-12 


$9,964 


116 


$86 


Rating based 
on participant 
survey 




+ 



Rating is expressed is contributing to any of the five AISD 
strategic objectives. 



+ 
0 

Blank 



er|c 



Positive, needs to be maintained or expanded 
Atol significant, needs to be improved ar ' modified 
Negative, needs major modification or replacement 
Unknown 
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Cost is the expense over the regular District per student 
expenditure of about $4,000. 

0 So cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct coit7, but under $500 per student 
$$$ Major direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 
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DEFINITIONS 



At r isk - In AISD, a student in grades 7-12 is considered at risk of dropping out if the student falls into one 
of 22 risk categories. 

Cost - The total cost of the program, regardless of funding source. The cost of a program is above and 
beyond the cost of the regular educational program. In reporting costs, ORE standardly uses appropriation 
or budget, not expenditure. Some programs have capital outlay costs, e.g., for computer equipment in a 
lab. These costs are shown as "investment cost," i.e., the initial cost of equipment and other items to get 
the program going. "Operating cost" is the annual cost to keep the program functioning after large initial 
outlays have been made. Cost figures are rounded to the nearest dollar. 

Cost/effect - "Cost per student" or "cost" (for dropout prevention programs) divided by "effect." 
"Cost/effect" is the annual cost for one month's extra achievement gain above that attributable to the regular 
instructional program. 

Cost-effectiveness (C/E) analysis - A type of cost analysis concerned with the evaluation of alternatives 
according to both their costs and their effects with regard to producing some outcome or set of outcomes. 
In C/E analysis, a measure of cost is divided by a measure of effectiveness. This analysis is distinguished 
from other cost-effectiveness analyses by the measure used as the denominator. In cost-benefit (C/B) 
analysis, by comparison, the denominator is benefit expressed in dollars. 

Cost per student - "Cost" divided by "number of students served." Service may have been provided to 
others besides students, e.g., teachers trained with Title II monies. In these instances, cost per participant 
should be understood. "Cost per student" is the numerator in the cost/effect calculation. 

Cost Rating - A rating scale is supplied by which the relative cost of programs can be broadly gauged. The 
$500 figure is an arbitrary selection based on experience. 

Cost is the expense over the regular District per-stiident expenditure of about $4,000. 

0 No cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 
$$$ Major direct cots for teachers, staff, and/or equipment 
in the range of $500 per student 

Dropout - A student is reported as a dropout for a school year if the individual is absent for a period of 30 
or more consecutive school days without approved excuse or documented transfer, or fails to reenroll by 
September 15 of the following school year without completion of a high school program. See "predicted 
dropout rate" and "obtained dropout rate." 

Dropout risk probability - Based on the risk factor associated with the student's membership in one of 22 
different risk categories. See "risk category" and "risk factor." 

The probability that a student will drop out is based on the actual percentage of students in that risk 
category who have dropped out in the past. For example, if 42.66% of the students in risk category #12 
dropped out the previous year, current-year students in that risk category would be assigned a dropout risk 
probability of 42.66. 

Effect - There are two measures of "effect." One is an achievement measure based on standardized test 
scores, and the second is a dropout prevention measure. All programs ultimately need to be held to the 
student achievement outcome criterion, even dropout and drug prevention programs. Like cost, the effect 
of a program, if any, is above that of the regular instructional program. 
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The ROSE residual (difference between predicted and obtained score) is the measure of achievement effect, 
unless the participants make up a disproportionate percentage of the comparison group. If the program 
participants do make up a disproportionate part of the comparison group, another standard for comparison 
was selected. 

Options other than ROSE residuals inciude: 

• Actual gain expressed in grade equivalents, 

• National norm gain residual, die difference between observed gain and an expected gain of 1 .0 GE 
per year on the average, and 

• AISD gain residual, the difference between observed gain and the average gain in the District. 

For a program like DARE, for example, where all the 5th- and 7th-grade students are in the program, the 
only comparison available is the national norm. 

"Disproportionate" is defined as the program students making up 25% or more of the AISD students at that 
grade or achievement level. 

Achievement effect is expressed as a number greater than one (1). A GE gain of three months, for 
example, is expressed as 3.0, instead of 0.3. 

The ROSE (residual) or dropout measure (predicted minus obtained rate) is used as the effect for those 
programs for which these measures can be obtained. For other programs, a 
+ /-/0/blank rating is assigned on the same basis as in past years' ORE reports. 

In the absence of a ROSE residual for the Composite test, the mathematics and reading residuals are 
averaged. 

The measure of dropout effect is the "number of predicted dropouts who stayed in school," i.e.. the number 
who did not drop out who were predicted to drop out. 

Funding source - Local, external, or both. External funding may be grant or other monies from other 
governmental entities or private organizations. 

Grade s - The grade levels served by the program. Analyses are based on the grade levels for which 
measures are available. For example, although a program may serve grades K-6, districtwide achievement 
test scores are not available for kindergarten. 

Level of service - Generally reported in one of three categories--(l) hours per week, 
(2) hours per day, or (3) full year--but may be more descriptive than quantitative. 

Number of students served - May be enrollment in the program or the definition used ir< the evaluation last 
year. Not all programs serve s'udents. In these instances, "number served" refers to participants. 

Obtained dropout rat e - For a program or group, the actual percentage of students who dropped out. 

Predicted dropout rate - For a program or group, the sum of the dropout risk probability for each student in 
the group divided by the number of students in the group (N). 

See "dropout risk probability," "risk category," and "risk factor." 

For example, if the total of the students' risk factors for 90 students served by a dropout prevention 
program were 3,333.80, the predicted dropout rate woulu be 37.042, or 37.0% (3,333.80/100 = 33.338 = 
33). IN other words, of 90 students served, 33 (37.0%) would be predicted to drop out based on their 
dropout risk probabilities. 
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The number of students predicted to drop out is not equivalent to the number of at-risk students 
because not all students who are at risk drop out, nor ;ire all the students who drop out identified as at risk. 

Predicted number of dropouts - For a program or group, the sum of the dropout risk probability for each 
student in the group divided by !00. 

See "predicted dropout rate." 

Program - Includes any special activity customarily thought of as a program. Some programs, e.g., Chapter 
2, have multiple program components. Programs often have separate budgets. 

Rating - A rating is supplied both for programs for which cost-effectiveness information can be provided 
and for programs about which ORE staff have an informed opinion based on evaluation information. In the 
former case, all programs which have a positive effect-defined as 0.1 GE (1 month's gain in grade 
equivalents) or better-will have a + rating (Because the cost-effectiveness ratio grows enormous the 
closer to zero effect size gets, it is impractical to report sizes smaller than 0.1 GE). In the case of 
programs for which ORE does not have cost-effectiveness information but does have sufficient evaluation 
information for an informed opinion, the rating scale used in the program effectiveness summary pages in 
last year's ORE final reports is applied: 

Effect is expressed as contributing to any of the five AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown, may have positive or negative impact on other indicators; however, impact on the five 
AISD strategic objectives is unknown. 

Risk category - One of 22 used to identify and track at-risk secondary (grades 7-12) students. ORE 
extended the four state-mandated criteria to pinpoint differential dropout rates. Greater percentages of 
students in some risk categories drop out than in other risk categories. Additional, optional criteria for 
identifying at-risk students have been specified by the State, e.g., sexual, physical, or psychological abuse, 
living in a residential treatment facility, and being homeless. However, AISD does not maintain centralized 
files on students with these characteristics. Therefore, ORE does not use these criteria to identify at-risk 
students. 

Seethe 1991-92 at-risk report (O n JE Publication No. 91.41) for definitions of the secondary risk categories. 

Risk factor - For a given risk category, the percentage of students in that risk category who dropped out. 
Expressed as a rate, the risk factor is a two decimal-place numeral. For example, if 45.75% of the students 
in a particular risk category dropped out, the risk factor for a student in that category would be 45.75. In 
other words, a student in this risk category would have almost a 50-50 chance of dropping out. 
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METHOD 



ORE has conducted and reported cost analyses for a number of years (see "References"). In 1992-93, 
ORE embarked on a new venture: cost-effectiveness analysis. Over a period of months, ORE staff engaged 
in considerable discussion about how cost-effectiveness should be calculated and how cost-effectiveness 
information should be integrated into ORE'S annual report to AISD's Board of Trustees about program 
effectiveness. A first-person account of how staff thinking evolved and what decisions were made is 
detailed in "Notes on Cost Effectiveness," ORE Publication Letter 92. D. The result of that thinking is 
contained in What Works, And Can We Afford It? Program Effectiveness in A1SD, 1991-92, ORE 
Publication Number 91.43. 

The methodology developed for assessing the cost-effectiveness of programs was applied and extended in 
ORE evaluations of 1992-93 programs. The following is a brief exposition of ORE's method in performing 
cost-effectiveness analyses on AISD programs. See "Definitions" and "Notes" for additional information. 

Following Levin (1983), cost-effectiveness is defined as cost divided by effect : 
Cost/Effect (C/E) 

Cost was defined, per earlier ORE research (see Wilkinson, 1985), as a program's appropriation (i.e., 
budget). Cost was taken to include all funding for a program, regardless of source. Effect was defined 
either as (1) achievement, (2) not dropping out, or (3) not using drugs. 

Definitions : Cost = appropriation (budget) 

Effect = achievement, OR 

not dropping out, OR 

not using drugs 

The achievement measure of effect was operational ized as the residual (i.e., difference) between the 
achievement of the program students and some standard or expectation for their achievement. A standard 
against which to compare is necessary to distinguish between the effect of the special program and the effect 
of the students' regular instructional program. Residual was defined as the difference between predicted and 
obtained scores, expressed in grade equivalents (GE's), from either the Norm-referenced Assessment 
Program for Texas (NAPT) or the Iowa Tests of Basic Skills (ITBS), both norm-referenced, standardized 
achievement test batteries. Three different residuals were identified: (1) average ROSE residual, (2) 
national norm gain residual, and (3) AISD gain residual. 

Definitions : Achievement = Average ROSE residual, 

OR National norm gain residual, OR 
AISD gain residual 

Residual = The difference between predicted and obtained 

score; for NAPT/ITBS, expressed in grade equivalents 
(GE's) 

Average ROSE residual = The average of the residuals from ROSE, on the reading 

and mathematics tests or the reading test alone, across 
grade levels, expressed in grade equivalents (GE's) 
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National norm gain residual = The difference between observed gain and an expected 

gain of 1.0 GE per year on the average 

AISD gain residual = The difference between observed gain and the average gain 

in the District, in GE's 

ROSE, the Report on School Effectiveness, is a series of regression analyses that answer the question, 
"How do the achievement gains of a school's students compare with those of other AISD students of the 
same previous achievement levels and background characteristics?" ROSE predicts achievement scores for 
the group of students who have both pre- and posttest scores on the ITBS, or the NAPT, depending on 
grade level and year of administration. Predictions are based on: 

Previous achievement level 
Sex 

Ethnicity 
Age 

Low-income status 
Family income 

Desegregation status of the school attended 
Whether or not the student was a transfer student 
Pupil-teacher ratio for school and grade 

The predicted scores are then compared with the students' actual scores. The difference between the 
predicted and actual scores is called the ROSE residual score, which is based on a GE score scale. If 
students' ROSE residual scores are far enough above or below zero to achieve statistical significance, they 
are said to have either "exceeded predicted gain" or to be "below predicted gain." Nonsignificant residual 
scores are classified as "achieved predicted gain." For more information about ROSE, see Paredes (1991). 

ORE'S GENeric Evaluation SYStem (GENESYS) produces, among other things, a Report on 
Program Effectiveness (ROPE). ROSE and ROPE are very similar, the major difference between them 
being that ROSE evaluates schools and ROPE evaluates programs. Most of the GE's used in calculations of 
achievement effect calculations were obtained from ROPE analyses produced by GENESYS. GENESYS also 
produces, for each program run, counts of the number of students predicted to drop out and the number who 
dropped out (see below). For more information about GENESYS, see Ligon and Baenen (1989) and 
Wilkinson and Spano (1990). 

The dropout prevention measure of effect was operational ized as the difference between the number of 
students in a program predicted to drop out and the actual number of students who dropped out. 

Definition: Not dropping out = The difference between the number of students predicted to drop out, 

based on their at-risk category, and the actual number of dropouts 



The drug prevention measure of effect was operationalized as the difference between the recent use of an 
illicit substance by program participants and by students in the District overall. 

Definition : Not using drugs = The difference between the recent use of an illicit substance by 

program participants and by students in the District overall. 
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The measure of students prevented from using drugs was based on self-reported use of alcohol, tobacco, and 
other illicit substances on the Student. Alcohol and Other Drug Use Survey, administered to AISD students 
in grades 4-12 in April 1993. The survey included items about the students' recent use of illicit substances. 
For students in grades 4-5, recent use is defined as use within the past school year; recent use by students in 
grades 6-12 is defined as use within the past 30 days. Students were also asked about their participation in 
Drug-Free Schools programs. The rate of recent use of any illicit substance was calculated for program 
participants and for the District as a whole. The number of students prevented from alcohol and other drug 
use reflects the difference between recent use by program participants and overall recent use by the entire 
sample, multiplied by the total number of students served by the program. 

Cost-effectiveness was operationalized as (1) cost per student divided by achievement effect, expressed in 
GE's, or (2) cost of the program divided by dropout prevention effect (predicted minus actual dropouts), or 
(3) cost of the program divided by drug prevention effect (average rate of drug use in the District minus the 
rate of use for program students times the number of students served by the program). 

Definitions : Cost/Effect = Cost per student/achievement effect, OR 

Cost for the program/dropout prevention effect, OR 

Cost for the program/drug prevention effect 

The cost-effectiveness ratio, expressed in dollars, which results from this division is a measure of the cost- 
effectiveness of a program, i.e., the amount of effect for monies expended, and because a common effect 
Tieasure was used as the denominator among like programs, programs' cost-effectiveness can be compared. 

Definition : Cost/Effect = Cost-effectiveness ratio (in dollars) 

Effect ratings were provided for programs (1) for which cost-effectiveness ratios could be calculated and 
(2) for which cost-effectiveness could not be calculated but about which other evaluation information was 
available. The ratings were based on the same scale which ORE had used four times previously. 



Definitions : Ratings: Same scale as in February 1993 and February 1993 program 

effectiveness charts; same as in ORE'S 1991-92 and 1992-93 final 
reports: 

Effect is expressed as contributing to any of the 5 AISD strategic 
objectives: 

+ Positive, needs to be maintained or expanded 

0 Not significant, needs to be improved and modified 

Negative, needs major modification or replacement 
Blank Unknown, may have positive or negative impact on 

other indicators; however, impact on the five AISD 

strategic objectives is unknown. 

Cost is the expense over the regular District per-student expenditure 
of about 4,000. 
0 No cost or minimal _ost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 
$$$ Major direct costs for teachers, staff, and/or 

equipment in the range of $500 per student or & more 
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Definitions : Effect Rating = + = Positive achievement gain, OR 

Number of students who actually dropped out was less 
than the number who were predicted to drop out, OR 
Rate of drug use by program students was less than the 
rate of use by students districtwide, OR 
Positive opinion, based on other indicators, such as 
survey results, lower retention, or other success 

0 = Achievement gain less than 1 month, OR 
Neutral opinion 

- = Negative opinion, OR 

Number of students who actually dropped out exceeded 
the number who were predicted to drop out, OR 
Rate of drug use by program students was greater than 
the rate of use by students districtwide 

Blank = Insufficient information 

Example #1 shows the cost-effectiveness computations for a program where ROSE residuals were used as 
the achievement effect measure. Example #2 shows the computations for a program where the AISD gain 
residuals were used as the effect measure. Example #3 shows the computations for a program using a 
dropout prevention effect measure. Example #4 shows the computations for a program using a drug 
prevention effect measure. 
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RATING 


+ 


EFFECT 
(in months) 


R: 11.0 
M: 7.0 
Avg.: 9.0 


COST 
PER 
STUDENT 


51,341 


NUMBER OF 
STUDENTS* 
SERVED 


00 

s 


ALLOCATION 
(COST) 


5815,604 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


5149 


PROGRAM 


Secondary Program 
Funding Source: Local 
Grades: 9-11 

Level of Service: Full year 
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RATING 


+ 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


$984 


PREDICTED 
DROPOUTS 
WHO STAYED IN 
SCHOOL 
(EFFECT) 


SO 
OS 

00 
CM 


NUMBER OF 
DROPOUTS 

Predicted Obuincd 


134 6 


COST 
PER 

STUDENT 


S940 


NUMBER OF 

STUDENTS* 

SERVED 


At risk: 134 
Total: 134 


ALLOCATION 
(COST) 


S26.0OO 


PROGRAM 


Newcomers Program 
Funding Source: External 
Grades: 9-12 
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